Hardware Usage Report

What is a Hardware Usage Report?

The Hardware Usage report provides a view of the Collibra DQ jobs that have consumed the most hardware.

You can generate a Hardware Usage report from the Reports page.

Steps

  1. Sign in to the Collibra DQ instance and click the reports icon Reports icon in the left navigation pane.
    The Reports page opens.

  2. From the Reports page, click the Hardware Usage Report link.

  3. For x-axis, select one of the following options from the dropdown:
    • num_executors
    • executor_memory
    • total_memory
    • total_cores
    • executor_cores
    • avg_row_count
    • dataset

  4. For y-axis, select one or more of the following options:

    • num_executors
    • executor_memory
    • total_memory
    • total_cores
    • executor_cores
    • avg_row_count
    • dataset

  5. For the type of chart you want to display, select one of the following options from the drop-down:

    • Line
    • Area
    • Column
    • Scatter

  6. Navigate the pages of your report by clicking the Previous and Next pagination buttons, located bottom-right of the columns.

Sample SQL query

You can use the following sample SQL query for a Hardware Usage report.

Copy 
with most_current_dataset_scan as ( select dataset, avg(rc)::Int as avg_row_count from dataset_scan group by dataset )

     select opt.dataset, num_executors, executor_cores, executor_memory, (executor_cores * num_executors) total_cores,
            (NULLIF(regexp_replace(executor_memory, '\D', '', 'g'), '')::numeric * num_executors) as total_memory,
            ds.avg_row_count
     from opt_spark opt, most_current_dataset_scan ds
     where opt.dataset = ds.dataset
     order by total_cores desc limit 75